27/06/2024 - StoRM migration + ingestion service as task

Finished ingestion service as task in rucio task manager (gave to JC to add to argocd, but image is done).

Migrated StoRM. Works, but still need to port data across. Having problems with volume attachments for rse data volume.

26/06/2024 - Ingestion node requirements

Continued drafting ingestion node requirements.

25/06/2024 - StoRM migration + ingestion node requirements + ingestion service as task

Continuing converting recipe below into helm chart but cloud went down for day.

Made confluence page with requirements for ingestion nodes.

Started looking at running the ingestion service as part of a task in rucio task manager.

24/06/2024 - StoRM migration

Converting recipe below into helm chart.

19/06/2024 - Ingestion service and StoRM migration

Got ingestion service working.

StoRM now running on rocky9 machine. Would like to turn this into an image + helm chart with PV so can run it on cluster.

Rough recipe:

sudo yum install git maven java-11-openjdk.x86_64
sudo git clone https://github.com/italiangrid/storm-webdav.git /opt/storm-webdav
sudo chown -R rocky:rocky storm-webdav/
sudo mkdir -p /usr/share/java/storm-webdav/
sudo mv storm-webdav-server.jar /usr/share/java/storm-webdav/
sudo mkdir -p /etc/systemd/system/storm-webdav.service.d/
(copy /usr/lib/systemd/system/storm-webdav.service)
(copy /etc/systemd/system/storm-webdav.service.d)
sudo ln -s /usr/lib/systemd/system/storm-webdav.service storm-webdav.service
sudo systemctl daemon-reload
sudo mkdir -p /etc/storm/webdav
sudo useradd -ms /bin/bash storm
(copy /etc/storm/webdav)

sudo mkdir -p /var/log/storm/webdav/
sudo chown -R storm:storm /var/log/storm/

sudo mkdir -p /var/lib/storm-webdav/work
sudo chown -R storm:storm /var/lib/storm-webdav/

(copy /etc/grid-security/storm-webdav/)

sudo service storm-webdav start
sudo service storm-webdav status

18/06/2024 - Site Capabilities API + rucio task manager / ingestion service

Added persistence to site capabilities API.

Helmified ingestion service + added ability to add persistent volumes to rucio task manager

18/06/2024 - Rucio migration + ingestion service

Migrated Rucio namespace.

Made numerous changes to ingestion service and added docs (https://ska-src-ingestion-ska-telescope-57397dbf79eedf6757468cdbd1a13d7.gitlab.io/srcnet-tools-ingest.html):

• restricted namespace uploads to staging directory path (not just namespace in metadata) stopping arbitrary namespaces being set and circumventing permissions.
• added gitlab ci
• added volume mounted token path in init (allowing host to get/refresh access tokens for service)
• added elasticsearch credentials env vars
• added concept of persisted message stores (allowing messages to be queued/sent after processing), but not finished handling these
• introduced mandatory and nonmandatory decorators for process_message to aid persistance of messages

17/06/2024 - Monitoring migration

Finished migrating monitoring namespace. See https://confluence.skatelescope.org/display/SRCSC/Tech+Debt%3A+Migrate+PVs+to+use+Manila+storage+rather+than+Cinder

14/06/2024 - Monitoring migration

Got this to work by deleting PVs and PVCs (retained), but the storage class hasn't migrated. Although it's listed as manila on the server, the provisioner used is still Cinder.

Put monit back online while i work this out. Will need to backup/restore again.

13/06/2024 - Openstack outage

APIs down because they can't see the SKA IAM (doesn't route on the instance but DNS works). Can't log in to Openstack. Can't describe Velero backups (I could yesterday).

Something is awry at RAL.

12/06/2024 - HQ

Mostly just catching up at HQ.

11/06/2024 - Looking at issue with multi attachments on cinder

Installed manila. Installed velero.

Can hopefully take backups and restore with different storage class:

https://velero.io/docs/v1.13/restore-reference/#changing-pvpvc-storage-classes

Snapshotted indices yesterday. If this is tested tomorrow this should be OK as volumes are set to retain when unattached.

01/06/2024 to 10/06/2024 - Leave/sick